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PREFACE 


This  effort  was  conducted  by  the  Hying  Training  Division  of  the  Air  Force  Human 
Resources  Laboratory,  Williams  Air  Force  Base,  Arizona.  The  project  was  completed 
under  Work  Unit  11230234  entitled  Advanced  Instructional  Features  and  Methods  in 
ASPT.  The  work  unit  supports  project  1123,  Hying  Training  Development;  task  112302, 
Instructional  Innovations  in  Hying  Training.  These  efforts  further  support  AFHRL 
Planning  Objective  G03,  Specific  Goal  2,  Training  Methods  and  Media. 

Special  thanks  are  extended  to  Mr.  James  F.  Smith  for  his  helpful  comments  in  the 
preparation  of  this  report. 
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APPLICATION  OF  FLIGHT  SIMULATOR  RECORD/PLAYBACK  FEATURE 


t.  INTRODUCTION 

Mott  flight  simulators  now  in  use  incorporate  training  features  that  are  intended  to  promote  greater 
instructional  efficiencies  (Caro,  1979;  Hughes,  1979;  Isley  &  Miller,  1976).  One  such  training  feature  is 
referred  to  generically  as  “record/playback.”  This  feature  permits  the  simulator  to  reproduce  a  previously 
established  segment  or  phase  of  flight  without  direct  intervention  by  the  trainee  or  the  Instructor/Operator. 
These  events  include  motion  system,  primary  flight  controls  and  displays,  and  appropriate  sensor  displays. 
Alternative  applications  of  the  record/playback  feature  include  employing  it  either  for  the  purpose  of 
presenting  a  recorded  demonstration  or  for  replaying  student  performance  for  the  purpose  of  critique. 

Although  interest  has  been  expressed  in  capitalizing  upon  such  unique  training  capabilities  (USAF, 
1978),  the  evidence  suggests  that  simulators  continue  to  be  used  as  “substitute”  aircraft.  Limitations 
inherent  in  the  use  of  operational  equipment  as  training  devices  continue  to  be  imposed  upon  simulator 
based  training.  The  record/playback  feature,  for  instance,  continues  to  be  used  only  in  the  most 
rudimentary  fashion  to  provide  a  duplicate  of  the  live  demonstration  as  provided  in  the  actual  operational 
equipment.  Rarely  is  the  use  of  record/playback  incorporated  with  other  features  (for  example,  freeze  and 
in-flight  condition  store)  to  achieve  a  unique  instructional  capability. 

Despite  the  literature  on  the  educational  use  of  video-taping  for  providing  instructional  feedback 
(Baron,  1969;  Carre,  1973;  Kraft,  1973;  Pease  &  Damron,  1974;  Sanders,  1969),  unequivocal  evidence  still 
does  not  exist  upon  which  to  base  a  rationale  for  the  instructional  use  of  an  automated  record/playback 
feature  in  flying  training  simulation.  Since  the  use  of  demonstrations  and  performance  replays  both 
constitute  instructional  events  which  subtract  from  the  time  available  for  student  practice,  it  is  imperative 
that  their  content  and  placement  in  the  learning  sequence  be  optimized.  In  automated  approaches  to  flying 
training  simulation,  for  example,  where  interaction  with  a  live  instructor  may  be  absent  and  where 
interaction  in  terms  of  feedback  characteristics  of  the  device  may  be  limited,  the  use  of  these  features 
becomes  even  more  critical. 

The  present  study  created  a  training  setting  where  instructional  material  was  presented  primarily 
through  pre-recorded  material  and  where  feedback  was  limited  to  an  overall  performance  “score.” 
Instructional  conditions  were  arranged  so  as  to  address  the  following  questions: 

1 .  For  the  particular  task  selected  for  study,  does  periodic  use  of  performance  playback  result  in 
more  rapid  acquisition  than  use  of  the  equivalent  amount  of  time  for  additional  student  practice? 

2.  For  the  particular  task  selected  for  study,  do  subsequent  exposures  to  the  original  demonstration 
result  in  more  rapid  acquisition  than  use  of  an  equivalent  amount  of  time  for  additional  student  practice? 


B.  METHOD 

Subjects.  Fifteen  USAF  Undergraduate  Pilot  Training  (UPT)  students  assigned  to  Williams  AFB, 
Arizona,  served  as  subjects.  All  students  serving  as  subjects  were  still  at  a  “pre-aerobatic”  phase  in  the  T-37 
portion  of  the  UPT  syllabus.  No  subjects  had  previous  experience  with  any  aerobatic  mmeuver  or  with  any 
of  the  major  component  parts  of  the  doverieaf  maneuver  flown  in  the  present  study. 

Apparatus.  The  study  was  conducted  on  the  Advanced  Simulator  for  Pilot  Training  (ASPT)  located  at 
the  Hying  Training  Division  of  the  Air  Force  Human  Resources  Laboratory,  Williams  AFB.  Technical 
references  for  the  device  are  found  in  Gum,  Albery,  and  Battnger  (1975)  and  in  Rust  (1975).  Provisions  for 
force  cueing  (e.g.,  platform  motion,  G-seat,  and  G-suH)  were  not  used.  A  computer-generated  visual  scene 
of  the  Williams  AFB  training  environment  wu  presented  via  ASPTs  seven  36-inch  monochromatic 
cathode-ray  tubes  placed  around  the  cockpit  giving  the  pilot  +1 10  degrees  to  —40  degrees  vertical  cueing 
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and  ±150  degrees  of  horizontal  cueing.  The  aerodynamic  models  driving  the  simulator  were  those  of  the 
T-37  aircraft. 


Procedure 

Preflight  Briefing.  Following  a  videotaped  introduction  to  the  purpose  of  the  study  which  included  a 
5  minute  preflight  briefing  on  performance  of  the  cloverleaf  maneuver,  each  student  entered  the  simulator 
and  witnessed  a  recorded  demonstration  of  the  task  to  be  learned,  i.e.,  the  first  two  leaves  of  a  cloverleaf 
maneuver  (see  Figure  1).  The  demonstration  included  all  visual  cues  associated  with  performance  of  the 
task  as  well  as  all  instrument  readings  and  stick  and  throttle  positions.  The  simulation  also  included 
recorded  aural  narration. 


Figure  1.  Flight  path  of  experimental  maneuver 
(darkened  area). 

Instructional  Conditions.  The  condition  under  which  a  student  practiced  the  task  was  determined  by 
assignment  of  the  student  to  one  of  three  instructional  groups  (see  below).  Regardless  of  group  assignment, 
all  students,  following  the  initial  demonstration,  performed  two  leaves  of  a  cloverlead  maneuver  for  four 
blocks  of  three  practice  trials  each  (i.e.,  each  student  practiced  12  complete  repetitions  of  the  task).  The 
three  instructional  conditions  are  described  below.  Five  students  were  assigned  to  each  of  the  three 
different  instructional  conditions: 

1.  “Demo”  Condition.  Between  each  successive  block  of  trials,  students  in  the  Demo  condition 
witnessed  the  original,  full  demonstration  of  the  cloverieaf  maneuver  with  recorded  instructor  pilot  voice 
instruction. 
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2.  " Replay  ”  Condition.  Between  each  successive  block  of  trials,  students  in  the  Replay  condition 
witnessed  a  replay  of  the  immediately  preceding  trial.  During  the  performance  replay,  narration  was 
provided  by  an  instructor  seated  in  a  second  cockpit  “slaved”  to  that  of  the  student’s. 

3.  "Practice"  Condition.  Between  each  successive  block  of  trials,  students  in  the  Practice  group  used 
the  available  time  to  perform  an  additional  practice  trial. 

Scoring  and  Performance  Measurement.  Other  than  the  performance  feedback  contained  in  the 
instructor’s  narration  accompanying  the  replay  in  the  Replay  condition,  the  only  feedback  provided  to  the 
student  pilot  was  an  overall  performance  rating  (Unsatisfactory,  Fair,  Good,  Excellent)  given  by  an 
instructor  located  in  a  second  simulator  cockpit  “slaved”  to  the  student’s  cockpit.  The  instructor’s  cockpit 
provided  all  instrument  readings  and  visual  cues  available  to  the  student.  Scoring  criteria  used  by  instructor 
pilots  were  those  set  forth  in  Air  Training  Command  Manual  514,  “Primary  Flying  Jet.”  Parameters 
sampled  by  the  system’s  automated  performance  measurement  capability  are  shown  in  Figure  2.  Also 
presented  in  Figure  2  are  the  tolerance  bands  for  each  measure  used  to  define  acceptable  performance. 
Scores  derived  from  the  automated  performance  measures  consisted  of  the  number  of  out-of-tolerance 
measures  for  each  of  the  two  leaves  of  the  maneuver.  The  scoring  profile  is  described  in  Appendix  A.  Since 
no  prior  studies  were  available  to  serve  as  guidance  in  the  derivation  of  a  performance  score  from  those 
parameters  sampled  by  the  system’s  automated  performance  measurement  capability,  two  methods  for 
generating  scores  were  employed. 

In  the  first  method,  all  parameters  shown  in  Figure  2  with  the  exception  of  those  for  Entry 
(predetermined  by  an  initial  condition)  and  those  for  Entry  into  the  second  leaf  (redundant  with  the  Exit 
value  obtained  on  the  first  leaf)  were  used  as  a  basis  for  a  score.  In  the  description  of  the  results  which 
follows,  the  error  score  derived  in  this  manner  is  referred  to  as  having  come  from  the  “Original  Criteria.”  In 
the  second  method,  only  those  parameters  indicated  in  Figure  2  by  an  asterisk  served  as  a  basis  for  a  score. 
A  score  derived  in  this  manner  will  be  referred  to  later  as  having  come  from  the  “Modified  Criteria.” 
Parameters  selected  for  inclusion  in  the  modified  criteria  were  those  judged  by  AFHRL/FT  instructor  pilots 
as  being  predictive  of  overall  performance. 


7 


Scortn*  Polnti 

for  Each  Lnf  Parameter  Scored  Tolerance  Band 


Entry 

1.  Airspeed  (KIAS) 

215-230 

2.  Heading  (Deg) 

Actual  Value 

45°  Pitch 

1.  Pitch  (Deg)* 

40-50 

2.  Airspeed  (KIAS) 

160-200 

3.  Average  GS* 

2.5— 3.5 

Peak  Pitch 

1.  Pitch  (Deg) 

60-70 

45°  Turn 

1.  Pitch  (Deg)* 

60-70 

2.  Airspeed  (KIAS) 

140-160 

3.  Bank  (Deg) 

85-95 

4.  Average  Gs 

1. 5-2.5 

Inverted 

1 .  Heading  Change  (Deg)* 

80-100 

2.  Bank  (Deg)* 

170-190 

3.  Airspeed  (KIAS) 

100-120 

4.  Average  Gs 

0.5-1. 3 

5.  Average  Roll  (Deg) 

minus  15  -  minus  21 

Exit  Value 

1.  Airspeed  (KIAS)* 

190-230 

2.  Heading  Change  (Deg)* 

80-100 

3.  Average  Gs 

2.0-3 .0 

Figure  2.  Automated  performance  measurement. 


UI.  RESULTS 


Subjective  Performance  Ratings.  On  the  average,  the  subjective  performance  ratings  given  by 
instructor  pilots  for  each  of  the  four  blocks  of  trials  for  each  of  the  three  experimental  groups  reflected 
little  change  in  student  performance  across  blocks  of  trials  regardless  of  the  conditions  under  which  the  task 
was  acquired.  Unsatisfactory  performances  at  the  outset  of  training  improved,  on  the  average,  only  from 
the  Unsatisfactory  to  the  Fair  to  Good  range. 

It  is  not  critical  for  the  present  study  that  instructor  ratings  appear  to  have  been  insensitive  to 
changes  in  student  performance.  It  is  important  to  point  out  here  that  the  cloverleaf  maneuver  was 
introduced  to  students  at  a  pre-aerobatic  stage  of  training.  These  students  had  none  of  the  prerequisite 
flying  skills  possessed  by  students  acquiring  the  maneuver  at  the  normal  point  in  the  syllabus.  Choice  of  the 
experimental  task  was  based  upon  (a)  the  point  in  UPT  syllabus  at  which  students  were  available,  (b)  the 
capability  for  automated  performance  measurement,  and  (c)  the  extent  to  which  the  task  was 
representative  of  other  complex  flying  maneuvers.  Performance  should  therefore  be  viewed  as  performance 
on  a  complex  psychomotor  task  representative  of  those  acquired  in  aerobatic  maneuvers.  It  would  be 
inappropriate  to  interpret  the  results  which  follow  in  terms  of  their  direct  application  to  the  specific 
operational  training  of  the  cloverleaf  maneuver  in  UPT. 

Automated  Performance  Scores.  Scores  based  upon  “Original”  and  “Modified”  criteria  are  presented 
in  Table  1  for  each  experimental  group  across  blocks  B1  to  B4.  While  the  correlation  between  error  scores 
defined  in  these  two  ways  was  found  to  be  statistically  significant  (r  =  .50,  p  <  .05),  the  magnitude  of  the 
relationship  argues  against  their  direct  interchangeability. 

Table  l.  Automated  Performance  Scores 


B1  B2  B3  B4 


Demo 

X 

s.d. 

Replay 

X 

s.d. 

Practice 

X 

s.d. 

Demo 

X 

s.d. 

Replay 

X 

s.d. 

Practice 

IX 

Modified  Criteria 

29.20 

27.80 

3.19 

1.30 

21.80 

18.20 

9.20 

4.87 

25.60 

21.60 

4.67 

4.39 

Original  Criteria 

22.80 

21.46 

3.65 

3.98 

16.60 

15.13 

5.26 

1.61 

19.33 

16.20 

4.38 

4.80 

23.80 

24.00 

6.38 

6.63 

14.20 

14.80 

1.64 

6.57 

19.80 

18.20 

3.49 

4.27 

18.20 

18.00 

6.46 

4.68 

10.87 

10.80 

3.48 

6.70 

14.73 

14.87 

5.45 

3.76 

Block  1  performances  were  compared  across  the  three  experimental  groups  for  each  of  the  two 
methods  used  in  defining  errors  to  determine  whether  significant  differences  in  group  performance  existed 
at  the  outset  of  training.  (Note  that  all  three  experimental  groups  performed  under  the  same  training 
conditions  during  Block  1.)  Regardless  of  whether  “Original”  or  "Modified”  criteria  served  as  the  basis  for 
the  error  score,  differences  between  groups  on  Block  1  did  not  exceed  differences  that  would  have  been 
expected  on  the  basis  of  chance.  Since  the  performances  of  all  groups  could  be  considered  comparable  at 
the  outset  of  training,  the  data  in  Figure  3  have  been  presented  in  terms  of  “Performance  (Errors)  Relative 
to  Block  1.”  Performance  is  displayed  on  the  vertical  axis;  blocks  of  trials  are  displayed  on  the  horizontal 
axis.  Differences  between  subjects  due  to  group  assignment  as  well  as  differences  in  performance  over  trials 
were  analyzed  separately  for  each  method  used  to  define  errors  (i.e.,  original  vs.  modified  criteria).  Data 
were  analyzed  by  a  split  plot  factorial  design.  Differences  between  subjects  due  to  assignment  to 
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'ORIGINAL  criteria 


BLOCKS  OF  TRIALS 


•mooified  CRITERIA* 


Figure  3.  Performances  across  successive  blocks  of  trials. 


experimental  condition  (Group  Effect)  were  significant  for  both  the  “Modified”  Group  (F(2.12)=9.2623,  p 
<  .05)  and  the  “Original”  Group  (F(2,12)=3.8230,  p  <  .05).  Significant  improvement  over  trials  (Trials 
Effect)  was  also  noted  for  the  “Original”  Group  (F(2,24)=4.3702,  p  <  .05)  and  the  “Modified”  Group 
(F(2,24)=3.6191,  p  <  .05).  There  were  no  significant  group  by  trial  interactions  noted.  Summary  tables  for 
the  analyses  are  presented  in  Appendix  B.  * 
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As  is  clear  from  Figure  3,  differences  between  the  three  approaches  used  to  teach  the  task  are 
difficult  to  distinguish  early  in  training.  One-way  ANOVAs  indicated  that  differences  which  had  begun  to 
emerge  by  Block  3  and  Block  4  favored  (at  the  p  =  .05  and  p  =  .06  level  for  the  “Modified”  and  “Original” 
criteria,  respectively)  the  Replay  condition  over  the  Demo  condition.  The  relationship  of  performances  in 
the  Practice  group  relative  to  those  of  the  Demo  and  Replay  groups  was  less  clear. 

It  is  important  to  point  out  that  when  performance  of  the  Practice  group  was  considered  as  a 
“baseline”  against  which  to  evaluate  the  effectiveness  of  the  Replay  condition,  no  significant  differences  in 
performance  were  noted.  Caution  must  be  taken  in  the  interpretation  of  the  present  findings  both  because 
of  the  small  sample  sizes  involved  and  because  of  the  type  of  task  and  experience  level  of  subjects  at  the 
time  the  particular  task  was  introduced.  It  is  important  also  to  note  that  although  Block  1  differences 
between  groups  were  not  statistically  significant,  differences  between  the  performances  of  each  of  the  three 
groups  throughout  training  were  very  close  to  differences  between  groups  at  the  outset  of  training.  While 
normalizing  performances  relative  to  Block  1  was  felt  to  be  justified  for  the  purpose  of  comparing 
improvements  in  performance  across  groups,  interpretation  of  the  data  when  transformed  in  this  way  must 
remain  guarded. 

Results 

The  results  of  this  study  indicated  the  following: 

1.  There  were  significant  differences  between  subjects  during  the  acquisition  of  the  task,  as  a 
function  of  the  instructional  condition  to  which  they  were  assigned. 

2.  Regardless  of  the  instructional  condition  to  which  subjects  were  assigned,  performances  showed 
significant  improvement  as  a  function  of  continued  practice  on  the  task. 

3.  By  the  end  of  training  (i.e.,  by  the  end  of  Block  4),  there  was  a  trend  toward  best  performance  in 
the  Replay  condition  and  poorest  performance  in  the  Demo  condition. 

4.  Performances  in  the  Practice  and  Replay  groups  did  not  differ  significantly  at  the  end  of  training. 


rv.  CONCLUSION 

Within  the  context  of  the  present  study,  there  appears  to  have  been  little  instructional  value 
associated  with  repeated  exposure  to  the  original  demonstration.  While  the  Replay  condition  produced 
terminal  performances  with  fewer  errors  on  the  average  than  the  Demo  condition,  it  must  be  pointed  out 
that  the  difference  between  the  Practice  and  Replay  groups  at  the  termination  of  training  was  not 
statistically  significant.  The  most  surprising  aspect  of  the  present  data  was  the  performance  of  students  in 
the  Practice  group  relative  to  that  of  students  in  the  so-called  “instructional”  conditions.  These  data  suggest 
that,  at  least  in  some  instances,  a  training  approach  (or  device)  that  provides  only  for  basic  performance 
feedback  (e.g.,  a  “score”)  might  be  equally  as  effective  as  one  with  provision  for  presenting  repeated  access 
to  recorded  demonstrations  and/or  replays  of  previous  performance.  In  terms  of  the  experimental  designs 
used  to  evaluate  the  effectiveness  of  automated  training  features  such  as  record/playback,  the  present 
design  emphasizes  the  desirability  of  a  no-instruction  (practice  only)  control  group  as  a  baseline  against 
which  to  evaluate  the  effectiveness  of  the  feature  in  question. 

The  training  implications  of  this  study  are  as  follows: 

1.  Repeated  presentations  of  a  recorded  demonstration  used  to  introduce  a  student  to  a  new  task 
appears  to  have  little  instructional  value.  The  training  time  consumed  by  repeating  the  demonstration 
would  appear  to  be  better  used  for  providing  the  student  with  additional  time  to  practice  the  task  for 
feedback  in  the  form  of  replays. 

2.  It  is  not  possible  to  state  unequivocally  that  use  of  the  playback  feature  leads  to  more  rapid 
acquisition  of  a  task  in  the  simulator. 
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3.  Complex  visual  flying  tasks  such  as  the  cloverleaf  may  be  acquired  in  the  simulator  with  a 
minimum  of  instructor  feedback.  The  implication  is  that  many  flying  tasks  may  lend  themselves  to  partial 
or  full  automation  in  the  simulator  because  of  the  self-generated  feedback  in  flying. 

4.  According  to  Caro  (1979),  the  extent  to  which  students  are  able  to  successfully  acquire 
performances  in  the  absence  of  a  highly  structured  training  environment  may  be  related  to  individual 
differences  (e.g„  field  independence-field  dependence).  Such  variables  should  be  investigated  as  simulator 
training  becomes  more  and  more  automated. 
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APPENDIX  A:  SCORING  PROFILE:  CLOVERLEAF 


SDS  Guide 


Sds  Key 

Data  Pool  Label 

Description 

Entry  VL1 

ASCORE31 

Entry  airspeed 

ASCORE  32 

Entry  heading 

45  Pitch  1 

ASCORE  33 

G  loading 

ASCORE  34 

Airspeed 

PK  Pitch  1 

ASCORE  35 

Peak  pitch 

45  Turn  1 

ASCORE  36 

Pitch  at  45  -deg  turn 

ASCORE  37 

Airspeed  45-deg  turn 

ASCORE  38 

Bank  45-deg  turn 

ASCORE  39 

Gs  45-deg  tum 

Invert  1 

ASCORE  40 

Heading  inverted 

ASCORE  41 

Bank  inverted 

ASCORE  42 

Airspeed 

ASCORE  43 

Gs  inverted 

A/M  GS  1 

ASCORE  46 

Ave  Gs  entry  to  45  pitch 

ASCORE  47 

Max  Gs  entry  to  45  pitch 

ASCORE  48 

Ave  Gs  45  pitch  to  45  turn 

ASCORE  49 

Max  Gs  45  pitch  to  45  tum 

ASCORE  50 

Ave  Ga  45  tum  to  inverted 

ASCORE  51 

Max  Ga  45  tum  to  inverted 

ASCORE  52 

Ave  Ga  inverted  to  level 

ASCORE  53 

Max  Ga  inverted  to  level 

AV  RL/G1 

ASCORE  44 

Ave  Rol  rate 

ASCORE  45 

Ave  rolling  Gs 

Exit  VL  1 

ASCORE  54 

Exit  airspeed 

ASCORE  55 

Exit  heading 

RoU  Dir  1 

EDGSCR1 

True  ■  left  rolls 

Fake  *  right  roll  -  error 

Smooth  1 

Scoring  Profile 

ASCORE  *1-20 

Smoothness  (SP  #2) 

Eight  second  delay  until  command  “CLEAR  TO  START.” 

Smoothness  starts  scoring  after  command. 

Airspeed  and  heading  captured  when  pitch  greater  than  10  degrees. 

Gs  and  airspeed  captured  at  45-degree  pitch. 

Peak  pitch  is  captured  during  roll  to  inverted. 

Pitch,  airspeed,  Gs,  and  bank  captured  at  45  degrees  of  turn. 

Heading,  bank,  airspeed  and  Gs  are  captured  when  pitch  less  than  zero. 

Exit  airspeed  and  heading  are  captured  when  pitch  greater  than  zero  and  smoothness  stops  . 
scoring. 

Max  Gs  and  average  Gs  are  collected  for  the  segments:  Entry  to  45-degree  pitch;  45-degree 
pitch  to  45-degree  turn;  45-degree  turn  to  inverted  and  inverted  to  level  flight. 

Average  roll  rate  and  roU  Gs  are  collected  during  die  segment  45-degree  pitch  to 
inverted. 

The  above  scoring  profile  is  performed  for  each  leaf. 

Scoring  is  selectable  to  score  two  or  four  leaves. 
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APPENDIX  B:  ANALYSIS  OF  VARIANCE  SUMMARY  TABLES 


ANOVA 


■•urea 

df 

ss 

MS 

F- Value 

P-Valua 

Errors  (“Original’ 

’  Criteria) 

Group 

2 

365.1130 

182.5565 

3.8230 

.0511 

Between 

12 

573.0236 

47.7520 

Treatment 

2 

91.3797 

45.6898 

4.3702 

.0235 

GxT 

4 

14.6962 

3.6741 

.3514 

.8411 

Within 

24 

250.9160 

10.4548 

Errors  (“Modified 

"  Criteria) 

Group 

2 

675.3333 

337.8667 

9.2623 

.004 

Between 

12 

437.7333 

36.4778 

Treatment 

2 

116.1334 

58.0667 

3.6191 

.0412 

GxT 

4 

10.1333 

2.5333 

.1579 

.9511 

Within 

24 

385.0665 

16.0444 

